Analysis and study on text representation to improve the accuracy of the Normalized Compression Distance

The huge amount of information stored in text form makes methods that deal with texts really interesting. This thesis focuses on dealing with texts using compression distances. More specifically, the thesis takes a small step towards understanding both the nature of texts and the nature of compression distances. Broadly speaking, the way in which this is done is exploring the effects that several distortion techniques have on one of the most successful distances in the family of compression distances, the Normalized Compression Distance -NCD-. The research carried out in this thesis can be divided into three parts. The first part, which corresponds to Chapter 5, experimentally evaluates the impact that several word removal techniques have on NCD-driven text clustering, with the aim of better understanding of both the nature of compression distances and the nature of textual information. This goal is accomplished by analyzing how the information contained in the documents and how the upper bound estimation of their Kolmogorov complexity progress as words are removed from the documents. One of the main conclusions that can be drawn from this analysis is that the clustering accuracy can be improved by applying a specific word removal technique. This distortion technique consists of removing the most frequent words of the language preserving the previous text structure. The second part of the thesis, which corresponds to Chapter 6, attempts to shed light on the reasons why the application of such a distortion technique can improve NCD-driven text clustering. The experimental results show that the maintenance of both the previous text structure and the remaining words structure have some relevance in the clustering behavior. The third part of the thesis, which corresponds to Chapter 7, applies the above mentioned distortion technique to NCD-driven document search. The application of compression distances to document search is not trivial due to the fact that they do not commonly perform well when the compared objects have very different sizes. An NCD-based document search engine that deals with that drawback by using passage retrieval, is used in the third part of the thesis. The results show that the search accuracy can be improved by applying the distortion technique presented previously. Summarizing, one of the distortion techniques explored in the thesis has been found to be beneficial both in NCD-based document clustering and in NCD-based document search.